Skip to content

fix(codegen): clean up orphaned scaffold on cancelled/failed generation - #1252

Closed
groupthinking wants to merge 3 commits into
mainfrom
claude/determined-maxwell-9t5kou
Closed

fix(codegen): clean up orphaned scaffold on cancelled/failed generation#1252
groupthinking wants to merge 3 commits into
mainfrom
claude/determined-maxwell-9t5kou

Conversation

@groupthinking

@groupthinking groupthinking commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Canonical issue

Closes #1253.

Outcome

A video-to-software request that is cancelled or fails mid-scaffold no longer leaks a uvai_project_* temporary directory. The off-loop write hops wait for their worker thread to settle before propagating cancellation (so cleanup never races a live writer), and both the mkdtemp window and the write-plan window remove the scaffold before the error propagates.

Scope

  • Included:
    • src/youtube_extension/backend/code_generator.py
      • _run_offloop — shield-and-wait wrapper (mirrors _run_sync_rpc in services/cloud/cloud_tasks_queue.py); all three _apply_write_plan hops routed through it.
      • _make_scaffold_dir — owns the mkdtemp cancellation window: shields the worker and removes any directory it created before re-raising CancelledError.
      • generate_project — obtains the scaffold path via _make_scaffold_dir and removes the tree on cancellation/failure.
    • tests/unit/test_code_generator.py — cancellation (both windows) + failure cleanup tests, and _run_offloop unit tests.
  • Explicitly excluded: no change to generated output, file ordering, or error semantics on the happy path; byte-for-byte output equality from perf: batch project scaffolding disk writes off the event loop #1251 preserved.

Risk

  • Risk level: low
  • Failure mode: the defect fixed was a leaked temp directory on a cancelled/failed request. Cleanup is best-effort (shutil.rmtree(..., ignore_errors=True)) and cannot itself fail the request.
  • Rollback: revert this branch; behaviour returns to the perf: batch project scaffolding disk writes off the event loop #1251 baseline (functional, but leaks on cancellation).

Verification

Tied to head a124bde.

  • Focused tests — tests/unit/test_code_generator.py: 87 passed. Both cancellation windows have prove-fail-verified regression tests (each fails against the pre-fix path, passes now).
  • Wider sweep — 520 passed across the code-generator, video-processing, deployment, and AI-generator suites.
  • Lint — ruff check clean on both changed files.
  • Types — mypy adds zero new diagnostics (the 13 reported errors are all pre-existing on untouched lines).
  • All code CI green on the head (lint-python, lint-frontend, guards, validate, bandit, python-safety, npm-audit, gitleaks, Security Scan – python/javascript, dependency-review, CodeQL); Vercel preview Ready and Vercel Agent Review passed (VADE marked its finding ISSUE_RESOLVED).
  • agent-completion/truth-gate — see below
  • Final human review

Production evidence

Python-only change on the POST /api/v1/video-to-software request path (same endpoint as #1239/#1240/#1251). Not exercised by the Next.js Vercel preview; no runtime behaviour change on success — the new cleanup path is only reached on cancellation/failure. Vercel production remains READY.

Agent handoff

  • One canonical issue is linked (Cancelled video-to-software request leaks the scaffold temp directory #1253)
  • No competing PR implements the same issue
  • Acceptance criteria satisfied (both leak windows closed; both flagged independently by CodeRabbit and Vercel VADE and confirmed resolved)
  • Required checks pass on the current head — blocked only by agent-completion/truth-gate
  • Human decision requested only for the truth-gate/merge to protected main

Agent provenance

The remaining red check is agent-completion/truth-gate (invalid_payload — missing agent-lock provenance manifest). This branch is claude/*, so the gate classifies the PR as agent-authored and requires a frozen pre-dispatch intent plus a trusted terminal agent result registered through orchestration. I have deliberately not hand-filled that manifest to force the gate green: doing so would fabricate provenance the gate exists to verify, which repo policy forbids (REAL_MODE_ONLY; "do not weaken or impersonate the gate", per #810). Disposition of that gate and the merge to protected main are left to human sign-off.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UpREES1d2uL2cq4k7jiezV

groupthinking and others added 2 commits August 2, 2026 12:23
ProjectCodeGenerator performed every scaffolding filesystem call inline
inside async def bodies, so a blocked write parked the whole event loop
rather than just the requesting coroutine.

Move all 28 filesystem calls off the loop, batched into one
asyncio.to_thread hop per generator (O(1) hops instead of O(files)).
Content generation is pure in-memory string building and stays on the
loop; only the writes are offloaded.

Output is byte-for-byte identical, verified by loading the pre- and
post-change modules side by side and comparing SHA-256 digests of every
emitted file plus every returned dict across all three generators.

Refs #1250

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Offloading the scaffolding filesystem work to `asyncio.to_thread` (#1250)
introduced `await` points that did not exist when the writes ran inline.
A request cancelled mid-scaffold now raises `CancelledError` while the
worker thread keeps running, so `generate_project` can abandon a
`uvai_project_*` directory whose path it never returns and no caller ever
cleans up. Reported by CodeRabbit on the scaffolding-off-loop change.

Fixes:
- Add `_run_offloop`, mirroring `_run_sync_rpc` in cloud_tasks_queue: it
  shields the worker task and waits for it to settle before propagating
  cancellation, so higher-level cleanup never races a live writer.
- Route all four off-loop hops (mkdtemp + three `_apply_write_plan`
  batches) through it.
- Wrap the post-mkdtemp region so a cancelled or failed generation removes
  the orphaned scaffold directory (best-effort, off-loop) before re-raising.

Adds cancellation/failure-cleanup tests and `_run_offloop` unit tests.
Behaviour on the happy path is unchanged; 520 passed across the code
generator, video processing, deployment, and AI generator suites.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UpREES1d2uL2cq4k7jiezV
@vercel

vercel Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
v0-uvai Ready Ready Preview, v0 Aug 2, 2026 5:40pm

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

🔍 PR Validation

⚠️ Large PR detected (586 lines changed)

@github-actions github-actions Bot added the python label Aug 2, 2026
@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • [‘architecture-gap’, ‘bug’, ‘ci-cd’, ‘ci/cd’, ‘copilot-rabbit’, ‘documentation’, ‘duplicate’, ‘enhancement’, ‘frontend’, ‘github_actions’, ‘good first issue’, ‘help wanted’, ‘high-priority’, ‘invalid’, ‘javascript’, ‘ml-model’, ‘needs-triage’, ‘pipeline-critical’, ‘placeholder-code’, ‘priority:high’, ‘python’, ‘python:uv’, ‘question’, ‘styling’, ‘tests’, ‘v0’]

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2b848665-0ded-4a5a-9ac8-4dec308d4681

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA a124bde.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Agent Completion Truth Gate: BLOCKED

Reasons: invalid_payload

Machine-readable verdict
{
  "details": {
    "invalid_fields": [
      "policy.agent_login",
      "policy.run_id"
    ]
  },
  "reasons": [
    "invalid_payload"
  ],
  "verdict": "blocked"
}

Workflow evidence

Copy link
Copy Markdown
Owner Author

Status at head 2355d04

Code checks are green. lint-python, lint-frontend, guards, validate, bandit, python-safety, npm-audit, gitleaks, Security Scan – javascript, dependency-review, and Canonical issue and evidence all pass; the Vercel preview is Ready; test/build/trivy/Security Scan – python/coverage are still running and green so far. Locally: 86 focused tests and 520 across the code-generator / video-processing / deployment / AI-generator suites pass; ruff clean; mypy adds zero new diagnostics.

The one red check is agent-completion/truth-gate (invalid_payload). Its verdict flags issue.number, policy.agent_login, policy.run_id — i.e. a missing agent-lock provenance manifest. This branch is claude/*, so the gate classifies the PR as agent-authored and requires a frozen pre-dispatch intent plus a trusted terminal agent result linked through that manifest.

I am not hand-filling that manifest to turn the gate green. Doing so would fabricate provenance the gate exists to verify, which the repo's policy explicitly forbids (REAL_MODE_ONLY; "do not weaken or impersonate the gate", as noted on #810). Even a well-formed payload would then fall to missing_agent_result — the same governance blocker currently on #810, #1242, and #1251 — because the frozen-intent and trusted-result artifacts are produced by the orchestration/human sign-off path, not by this fix.

What this PR is: it resolves CodeRabbit's blocking cancellation finding on #1251 — a request cancelled mid-scaffold could leak a uvai_project_* directory once the writes moved off the event loop. The write hops now shield their worker and wait for it to settle before propagating cancellation, and generate_project removes the orphaned scaffold on cancellation/failure.

Next step is human: review, then decide the truth-gate/merge disposition to protected main. I won't force the provenance gate.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

@coderabbitai review

This PR resolves the blocking cancellation finding you raised on #1251: a request cancelled mid-scaffold could leak a uvai_project_* directory once the writes moved off the event loop. Please verify specifically that:

  1. _run_offloop shields the worker and waits for it to settle before propagating CancelledError, so cleanup never races a live writer;
  2. generate_project removes the orphaned scaffold on both cancellation and failure, before re-raising;
  3. happy-path output is unchanged (byte-for-byte equality from perf: batch project scaffolding disk writes off the event loop #1251 preserved).

Generated by Claude Code

@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '=== PR diff ==='
git diff --find-renames --unified=80 'origin/main...HEAD' -- \
  src/youtube_extension/backend/code_generator.py \
  tests/unit/test_code_generator.py

printf '%s\n' '=== implementation outline ==='
ast-grep outline src/youtube_extension/backend/code_generator.py --items all --view expanded

printf '%s\n' '=== test outline ==='
ast-grep outline tests/unit/test_code_generator.py --items all --view expanded

printf '%s\n' '=== relevant definitions and call sites ==='
rg -n -C 5 -P '\b(_run_offloop|generate_project|_apply_write_plan)\s*\(' \
  src/youtube_extension/backend/code_generator.py \
  tests/unit/test_code_generator.py

printf '%s\n' '=== output/equality assertions ==='
rg -n -i -C 4 'byte|equal|equality|expected|write.plan|generate_project' \
  tests/unit/test_code_generator.py

Length of output: 56777


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '=== locate focused test file ==='
fd -a -t f '^test_code_generator\.py$' .

printf '%s\n' '=== generate_project and off-loop helper ==='
nl -ba src/youtube_extension/backend/code_generator.py | sed -n '25,265p'

printf '%s\n' '=== current write plans ==='
nl -ba src/youtube_extension/backend/code_generator.py | sed -n '275,460p'

printf '%s\n' '=== baseline write sequences ==='
git show origin/main:src/youtube_extension/backend/code_generator.py \
  | nl -ba \
  | sed -n '175,445p'

printf '%s\n' '=== test names and cancellation/equality coverage ==='
fd -a -t f '^test_code_generator\.py$' . -x sh -c '
  printf "\n--- %s ---\n" "$1"
  rg -n -i -C 8 "mkdtemp|cancel|orphan|offloop|byte|equal|verbatim|write.plan" "$1"
' sh {}

Length of output: 1202


@groupthinking I found one blocking cancellation leak.

_run_offloop correctly shields each write worker and waits for it to finish before it re-raises CancelledError. This prevents _safe_rmtree from racing an active _apply_write_plan worker.

However, generate_project creates the cleanup scope only after this statement completes:

temp_dir = await _run_offloop(tempfile.mkdtemp, prefix="uvai_project_")

If cancellation occurs while the mkdtemp worker runs, _run_offloop waits until mkdtemp creates the directory and then raises CancelledError. temp_dir is not assigned, and the inner try/except that calls _safe_rmtree is never entered. The newly created uvai_project_* directory leaks.

Please make the mkdtemp hop own cleanup when cancellation occurs after directory creation, or otherwise retain its result long enough to remove it before propagating cancellation. Add a regression test that blocks mkdtemp, cancels the generation task, releases mkdtemp, and asserts that the created directory no longer exists.

The diff preserves the existing write order and uses equivalent content generation on the normal paths. I could not independently verify the focused test file because tests/unit/test_code_generator.py was not present in this checkout.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Comment thread src/youtube_extension/backend/code_generator.py Outdated
CodeRabbit and Vercel VADE both flagged a remaining leak in the previous
commit: the cleanup try/except began only after
`temp_dir = await _run_offloop(tempfile.mkdtemp, ...)`. If cancellation
landed while the mkdtemp worker was running, the shielded hop still created
the directory and then raised CancelledError before the assignment, so the
new `uvai_project_*` directory never entered the cleanup scope and leaked.

Add `_make_scaffold_dir`, which owns that window: it shields the mkdtemp
worker, and on cancellation removes any directory the worker created before
propagating CancelledError. `generate_project` now obtains the scaffold
path through it, so both the creation window and the write phase are
leak-safe.

Adds a regression test that blocks mkdtemp, cancels the generation task,
releases mkdtemp, and asserts the created directory is gone (prove-fail
verified: it fails against the prior leaky path). 87 focused tests pass;
ruff clean; mypy adds no new diagnostics.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UpREES1d2uL2cq4k7jiezV

Copy link
Copy Markdown
Owner Author

Fixed in a124bde. Both CodeRabbit and Vercel VADE independently flagged the same real leak: the cleanup try began only after temp_dir = await _run_offloop(tempfile.mkdtemp, ...), so a cancellation delivered during the mkdtemp hop created the directory and raised CancelledError before the assignment — the path never entered the cleanup scope.

The mkdtemp hop now owns that window via _make_scaffold_dir: it shields the worker and, on cancellation, removes any directory the worker created before propagating CancelledError. generate_project obtains the scaffold path through it, so both the creation window and the write phase are leak-safe.

Verification: added a regression test that blocks mkdtemp, cancels the task, releases mkdtemp, and asserts the directory is gone — prove-fail confirmed it fails against the prior leaky path and passes now. 87 focused tests pass; ruff clean; mypy adds no new diagnostics.

The agent-completion/truth-gate remains the only non-code blocker (provenance/human sign-off, as noted above) — unchanged by this commit.


Generated by Claude Code

@groupthinking

Copy link
Copy Markdown
Owner Author

Superseded by #1251, which carries both defects this PR fixes plus the off-loop conversion of the 28 scaffolding writes.

Full rationale and a side-by-side comparison are in #1251 (comment). Summarised: the two branches converged independently on the same shield-and-drain design derived from _run_sync_rpc, which is good corroboration. #1251 is kept because _discard_project_dir catches Exception whereas _safe_rmtree here relies on rmtree(ignore_errors=True), which is not total — a NUL byte in a path raises ValueError before the error callback is consulted. On the failure path that could mask the original exception.

Credit where due: this PR was ahead of #1251 on the generic-exception path, and that gap was real. It is now closed there in d178e210a.

Please reopen if #1251 stalls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cancelled video-to-software request leaks the scaffold temp directory

2 participants